Logsearch with Docker

Preparation

To get started you will need a running bosh.

Next step is setting up this repository

git clone https://github.sw.ge.com/cf-platform-services/docker-services-boshworkspace.git cd docker-services-boshworkspace bundle install Upload the BOSH release

Upload the Docker BOSH release as follows:

bosh target BOSH_HOST git clone https://github.sw.ge.com/cf-platform-services/docker-boshrelease.git cd docker-boshrelease bosh upload release releases/docker/docker-10.yml Deploy Docker Services

Initiate each new deployment with the following command:

bosh setup deployment It will prompt you for which service to deploy (or ALL services).

Then, it will prompt for AWS/OpenStack specific questions. See those sections below.

For example with bosh-lite/warden you will see something like:

WARNING: loading local plugin: lib/bosh/cli/commands/setup_deployment.rb Looking up 'cf-warden'...

  1. ALL
  2. ArangoDB 2.2
  3. Consul 0.3.1
  4. CouchDB 1.6
  5. Elasticsearch 1.3
  6. Etcd 0.4.6
  7. Logstash 1.4
  8. Memcached 1.4
  9. MongoDB 2.6
  10. MySQL 5.6
  11. NATS
  12. Neo4j 2.1
  13. PostgreSQL 9.3
  14. RabbitMQ 3.3
  15. Redis 2.8
  16. RethinkDB 1.14.0 Choose a service (or ALL): 7 bosh deployment deployments/my-neo4j21-services-warden.yml WARNING: loading local plugin: lib/bosh/cli/commands/setup_deployment.rb Deployment set to `.../.deployments/my-neo4j21-services-warden.yml'

bosh deploy ... Are you sure you want to deploy? (type 'yes' to continue): Type yes to continue with the deployment.

After BOSH finishes the deployment, the broker is not yet ready. The terminal will start polling for the broker. For a few minutes the VM will be downloading the 1 docker image per service.

Finally, you will be prompted to run a command like the one below. It will include the correct password:

cf create-service-broker docker containers PASSWORD http://cf-containers-broker.10.244.0.34.xip.io Once the command above works you can now enable your services to some/all organizations:

cf service-access cf enable-service-access Deploy on AWS VPC

To deploy any or all the docker services to your AWS VPC, run:

bosh setup deployment It will prompt you for the following and ultimately commence deployment of the VM:

select your target Cloud Foundry (or will show the only CF deployment if there is only one) select a docker service to deploy into a single VM (or 'ALL' if your single VM wishes to support all docker services) select an AWS instance type (list include 64-bit, paravirtual, with some ephemeral disk for docker containers) specify the persistent disk size (where all services' data is stored) specify the subnet ID (e.g. subnet-5d51d338) specify the subnet range (e.g. 10.10.5.0/24) OR specify the sub-range within a shared subnet (e.g. 10.10.5.16/30) confirm to commence deployment of the new BOSH deployment manifest The output will look similar to below:

$ bosh setup deployment WARNING: loading local plugin: lib/bosh/cli/commands/setup_deployment.rb Looking up 'cf-aws-tiny'...

  1. ALL
  2. Memcached 1.4
  3. MongoDB 2.6
  4. CouchDB 1.6
  5. NATS
  6. Redis 2.8
  7. Elasticsearch 1.3
  8. Neo4j 2.1
  9. Logstash 1.4
  10. Etcd 0.4.6
  11. Consul 0.3.1
  12. PostgreSQL 9.3
  13. MySQL 5.6
  14. RabbitMQ 3.3
  15. RethinkDB 1.14.0
  16. ArangoDB 2.2 Choose a service (or ALL): 2

Security groups: cf-0-vpc-fa2f849f

  1. m1.large (850 disk, 7680 ram, 4 cores)
  2. m1.xlarge (1690 disk, 15360 ram, 8 cores)
  3. c1.xlarge (1690 disk, 7168 ram, 20 cores)
  4. c3.large (32 disk, 3750 ram, 7 cores)
  5. c3.xlarge (80 disk, 7168 ram, 14 cores) ... Instance type: 1

Persistent disk volume size (Gb): 200

Subnet ID: subnet-5d51d338 No other deployments using same subnet Subnet CIDR range: 10.10.5.0/24 It will then automatically target the generated deployment manifest:

bosh deployment deployments/cf-containers-broker-memcached14.yml Deployment set to `.../.deployments/cf-containers-broker-memcached14.yml' And then attempt to deploy the new deployment manifest:

bosh deploy

WARNING: loading local plugin: lib/bosh/cli/commands/setup_deployment.rb Generating deployment manifest ...

Deploying

Deployment name: cf-containers-broker-memcached14.yml' Director name:bosh-vpc-fa2f849f' Are you sure you want to deploy? (type 'yes' to continue): Type yes to continue with the deployment.

After BOSH finishes the deployment, the broker is not yet ready. The terminal will start polling for the broker. For a few minutes the VM will be downloading the 1 docker image per service.

Finally, you will be prompted to run a command like the one below. It will include the correct password:

cf create-service-broker docker containers PASSWORD http://cf-containers-broker.10.244.0.34.xip.io Once the command above works you can now enable your services to some/all organizations:

cf service-access cf enable-service-access Re-using a subnet

On AWS VPC there is support in bosh setup deployment to reuse a subnet. It will create a valid BOSH deployment manifest networks subnet with reserved ranges to protect your other deployments.

If you choose a subnet that is already being used by another BOSH deployment (within the same BOSH) you will be prompted:

Subnet ID: subnet-5d51d338 Other deployments using same subnet 'subnet-5d51d338': cf-containers-broker-memcached14 Ctrl-C to cancel to choose alternate subnet, or... Enter range of IPs (CIDR format: 10.10.5.0/30): 10.10.5.16/30 Subnet range: 10.10.5.0/24 Subnet useful range: 10.10.5.16-10.10.5.19 Subnet reserved ranges: 10.10.5.2-10.10.5.15, 10.10.5.20-10.10.5.254 Confirm these subnet sub-ranges make sense and press ENTER. If they don't, Ctrl-C, repeat and enter valid CIDR above... In the example above the subnet subnet-5d51d338 is already being used by another BOSH deployment (it looks through all deployments in the target BOSH director).

The full range of the subnet is 10.10.5.0/24, so the user entered a sub-range of 10.10.5.16/30.

The terminal confirms that this maps to the range of IPs: 10.10.5.16-10.10.5.19

More importantly from a BOSH deployment manifest perspective, it shows the unusable/reserved ranges of IPs - everything else in the subnet that is not 10.10.5.16/30.

If this is correct, press ENTER.

The BOSH deployment will continue as previously described above.

Enabling security group access

It is possible that your Cloud Foundry is configured by default to disallow applications to communicate with your Docker services. This is true for the bosh-lite deployment, for example.

cf create-security-group docker extras/sg-bosh-lite.json cf bind-running-security-group docker Restart any applications to have them granted access to Docker services.